Multispectral Colour Constancy

Does extending the number of channels from the 3 RGB sensors of a colour camera to 6 or 9 using a multispectral camera enhance the performance of illumination-estimation algorithms? Experiments are conducted with a variety of colour constancy algorithms (Maloney-Wandell, Chromagenic, Greyworld, Max RGB, and a Maloney-Wandell extension) measuring their performance as a function of the number of sensor channels. Although minor improvements were found with 6 channels, overall the results indicate that multispectral imagery is unlikely to lead to substantially better illumination-estimation performance. Introduction Illumination-estimation methods for colour constancy have been generally based on analysing the RGB values from a 3channel colour image; however, the Chromagenic algorithm [1] is a recent exception in that it uses a total of 6 channels. The Maloney-Wandell [3] is another notable exception. It is defined for an arbitrary number of channels. It might be expected that the more channels, the better, but is this the case? In other words, will using a multispectral camera potentially lead to better illumination estimates than a standard color camera? We address this question by testing a number of illuminationestimation algorithms on 3-channel, 6-channel and 9-channel multispectral image data, and comparing their accuracy as the number of sensors and their spectral sensitivity functions are varied. Since there are many avenues to explore, we restrict our attention to synthetic multispectral images; however, the synthesis is based on real reflectance and illumination spectra and variations on real camera sensitivities. We test the Greyworld, Max RGB, Chromagenic, and Maloney-Wandell algorithms. In addition, we introduce a new modified version of the Maloney-Wandell algorithm called Subspace Testing that works surprisingly well. Illumination estimation plays a central role in colour constancy, automatic white balancing, and colour cast removal. For a scene that is illuminated by light of a spectral power distribution that remains constant, except for variations in intensity, throughout the whole scene, the goal of illuminationestimation is to recover the colour (up to a scaling for intensity) of that light as it would be seen by a camera in the camera’s RGB colour space when it is reflected off an ideal-white surface. Since intensity is not recovered, it is natural to use chromaticity space for which we use rgb-chromaticity defined as rgb with r=R/(R+G+B) etc. Once the illuminant chromaticity is known, a simple scaling of the colour channels derived from the rgb suffices to ‘white balance’ the image and remove any colour cast the image may have had. The paper is organized as follows. First, the error measures for performance evaluation are discussed. Second, the standard algorithms to be tested are described along with the changes required in extending them beyond 3 channels. Third, the Subspace Testing algorithm is introduced. Fourth, the procedures for introducing the additional camera sensitivity functions and for synthesizing multispectral images are described. Fifth, the results are given followed, sixth, by the somewhat surprising conclusion they lead to. Performance Measures The angle between the rgb of the actual illumination and the estimated rgb provides a good measure of the performance of an illumination-estimation algorithm and has been widely used [1][5]. We use it here as well, but in two new ways. The first is simply to extend the definition to N dimensions, where N is the number of channels, and measure the angle between two N-dimensional vectors. The second is to convert N-dimensional illumination estimates into the rgb chromaticity space of a given 3-channel camera and then measure the angle in 3-space. We used the sensor sensitivities of Sony DXC-930 for the conversion. The advantage of converting the results into rgb space is that it allows for fairer comparison across results achieved in spaces of different dimensions. Conversion also helps determine whether the additional information that comes from higher dimensions will help achieve better white balance for the purposes of displaying the results on an RGB monitor. The conversion from Ndimensions to 3 is done by identifying which illuminant from a database of known illuminants it is most similar to, and using that illuminant’s rgb as the conversion value. Associated with each illuminant in the database is its ‘colour’ in N-dimensions. The Ndimensional colour to be converted is compared to those in the database to find the one with the smallest angular difference. Since the illuminants in the database will not cover the Ndimensional space continuously some error is introduced through conversion. However, conversion also introduces an additional constraint on the estimated illumination; namely, the conversion constrains the results to fall into the space of known illuminants. Color-by-correlation [2] employs this constraint. We find here that it improves the performance of some other algorithms, MaloneyWandell in particular. For a set of test images, the following error measures are calculated and reported in the Tables below. Median Angular Error in Image Space (Image). This is a median of all angles between the illuminant estimate in Ndimensional space and the actual illuminant in the N-dimensional space. Median Angular Error in rgb Space (rgb). Some algorithms, for example Maloney-Wandell, can output rgb vector of the estimated illuminant directly, without using the lookup in the illuminant database. This measure is a median of all angles between illuminant estimate in rgb space (as output directly by the algorithm) and the actual rgb vector of the illuminant. Median Angular Error for Illuminant Estimates Converted to rgb Space (Lookup rgb). This measure is obtained by converting the N-dimensional vectors output by the algorithms into rgb space by the lookup in the illuminant database described above. The median of the angles between converted illuminant estimates (which are now in rgb space) and rgb vectors of the actual illuminants is computed. Mean of the Top 0.5% Angular Errors of Converted Illuminant Estimates in rgb Space (Max Lookup rgb). This measure is obtained similarly to the “Lookup rgb” measure. The N-dimensional illumination estimates output by the algorithms are converted to rgb space by lookup in the illuminant database as described above. The mean of the top 0.5% of the angles between converted illuminant estimates (which are now in rgb space) and rgb vectors of the actual illuminants is computed. This measure shows the magnitude of the largest errors produced by the algorithms. Median Error of the Lookup Conversion in the Image Space (Error Lookup) As mentioned above, the illuminant database used for conversion from N-dimensional image space into rgb space does not cover the N-dimensional space completely. This measure shows the median of angles between the illumination estimates produced by the algorithms and the closest match in the illumination database. Algorithm Summaries We include the Greyworld and Max RGB algorithms in our experiments since they bridge across results reported by different authors. For an RGB image, the Greyworld algorithm computes mean R, mean G and mean B values. These mean values are assumed to represent the illuminant colour. In N dimensions, the process is the same except carried out over N channels instead of 3. The Max RGB algorithm finds the pixel with highest R value, a potentially different pixel with highest G value and a pixel (which is potentially different from previous two) with the highest B value. These R, G and B values determine the answer. The process for an image with N colour dimensions is the same except using N channels. We also include results for the ‘do nothing’ algorithm, which simply assumes the scene illuminant is always white. The donothing angular error provides a measure of the variation in incident illumination and establishes a benchmark with which to compare other algorithms. The Chromagenic algorithm [1] is somewhat more complex than Greyworld and Max RGB. The Chromagenic algorithm requires a 6-channel image, but the 6 channels are related to each other in a specific way. One way to acquire a 6-channel Chromagenic image is to take two registered images of the same scene. The first is taken in the usual way using a standard colour camera with 3 RGB sensors. The second is taken using the same camera, but with a filter placed in front of its lens. Clearly, the second 3 channels are related to the first 3. The Chromagenic algorithm exploits the relationship between them. The Chromagenic algorithm aims to identify the scene illuminant as one member of a set of known illuminants. It requires a training set of images with numerous images taken under each different illuminant. During the training phase, the Chromagenic algorithm computes a linear mapping from filtered RGB responses to non-filtered RGB responses for each of the illuminants. The mapping is computed as follows [1]: Let Pj and Pj F denote Nx3 matrices of RGBs and filtered RGBs for N surfaces measured under the j-th of M training illuminants. The linear mapping Tj for j-th illuminant is computed as Tj = [ Pj ] Pj (1) (1) where + denotes pseudo-inverse. Given an input image under unknown illumination, the Chromagenic algorithm considers each of the training illuminants in turn as a candidate. The candidate illuminant whose linear mapping best maps filtered RGB responses of the input image to non-filtered RGB responses of the input image is returned as the estimate of the unknown illumination. The Chromagenic algorithm is extended to N-dimensions by simply dividing the set of channels into two sets. Two sets represent “filtered” and “unfiltered” image. The number of channels in the two sets need not to be the same. However, for Chromagenic algorithm to work successfully, the “filtered” channels may not be completely independent from the “unfiltered” channels. The algorithm then computes and uses a mapping from “filtered” to “unfiltered” channels as described above. The Maloney-Wandell algorithm [4][3] assumes that the subspace of reflectances of all surfaces is linear and of reasonably small dimension. Dimensionality of the reflectances, M, is assumed to be smaller than the number of sensors, N, used to obtain the image. Hence the sensor responses for the surfaces under a single illuminant fall within a linear subspace of the same dimensionality. The illuminant defines the specific subspace. The algorithm proceeds by fitting an M-dimensional hyperplane through the origin to the sensor responses from the N-channel input image. The orientation of the plane determines the illuminant under which the image was taken. The Maloney-Wandell algorithm produces spectral power distribution of the illuminant (up to a scaling). This is one of the algorithms that can output its illuminant estimate directly in rgb space. The spectral power distribution of the illuminant can be converted into N-dimensional images space (by using the spectral characteristics of the N sensors) or directly into rgb space (by using spectral characteristics of the RGB sensors). We also introduce a modification to the Maloney-Wandell algorithm called the Subspace Testing algorithm. Subspace testing begins with the Maloney and Wandell’s observation that the image colours (we extend the term ‘colour’ to N dimensions from 3) will lie in a linear subspace of the N-dimensional image input space. The modification is in the way in which the subspace is identified. Firstly, the Subspace Testing algorithm is trained on a set of illuminants with many images under each. For each illuminant, an M-dimensional subspace is fitted over the sensor responses from its entire collection of images. This hyperplane represents the gamut of possible colours (N-dimensional) under that illuminant. To identify an image’s illuminant, Subspace Testing—rather than fitting a subspace over the responses present in the image as Maloney-Wandell does—tests the fit of the image colours to each of the training-derived hyperplanes. In particular, the hyperplane for which the root mean square distance between it and the input image’s colours is smallest is selected. Since this hyperplane is from the training set, it has an illuminant associated with it. The colour of that illuminant is then output as the estimate of the colour of the image’s illuminant. Data Preparation All experiments are based on images synthesized using the reflectance spectra, illuminant spectra and spectral sensitivity functions contained in the Simon Fraser University dataset (http://www.sfu.ca/~colour/data)[6]. The dataset contains spectral distributions of 1995 surface reflectances and 287 illuminants. Of the illuminant spectra 87 are used for training. All 287 illuminants are used for testing. Multispectral images are synthesized for the case of 3, 6 and 9 sensor classes. For the 3-sensor case, we use the sensitivity functions Sony DXC-930 and normalize them to have equal total sensitivity. To extend the sensor set to 6, we add the same Sony DXC-930 RGB sensors shifted by a fixed wavelength towards the red part of the spectrum. To extend the sensor set to 9, we add the same camera sensors twice; one copy shifted towards red and the second towards blue. We experiment with different amount of shifting. Figure 1 shows the spectral sensitivity functions for 9 sensors based on shifts of ±16nm. 300 400 500 600 700 800 0 0.05 0.1 Figure 1. Multispectral sensitivity functions for the 9-sensor case. Sensitivity is plotted as a function of wavelength. The normalized Sony DXC-930 camera RGB sensors are the middle of each group with copies shifted by